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Top  Five  Systems  Engineering  Issues* 


•  Lack  of  awareness  of  the  importance,  value,  timing, 
accountability,  and  organizational  structure  of  SE  on 
programs 

•  Adequate,  qualified  resources  are  generally  not  available 
within  government  and  industry  for  allocation  on  major 
programs 

•  Insufficient  SE  tools  and  environments  to  effectively 
execute  SE  on  programs 

•  Requirements  definition,  development,  and  management 
is  not  applied  consistently  and  effectively 

•  Poor  initial  program  formulation 

*  Based  on  an  NDIA  Study  in  January  2003 
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Recap:  What  We  Have  Done  To 
Revitalize  Systems  Engineering 


•  Issued  Systems  Engineering  (SE)  policy 

•  Issued  guidance  on  SE  and  Test  &  Evaluation  (T&E) 

•  Integrating  Developmental  T&E  with  SE  policy  and 
assessment  functions  -  focused  on  effective,  early 
engagement  of  both 

•  Instituted  system-level  assessments  in  support  of  OSD 
major  acquisition  program  oversight  role 

•  Established  SE  Forum  -  senior-level  focus  within  DoD 

•  Working  with  Defense  Acquisition  University  to  revise 
SE,  T&E,  and  enabling  career  fields  curricula 

•  Leveraging  close  working  relationships  with  industry  and 
academia 
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Systems  Engineering  Revitalization 

Framework 


Driving  Technical  Excellence  into  Programs! 
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Driving  Technical  Rigor  Back  into 
Programs  “Portfolio  Challenge” 


•  Defense  Systems  was  tasked  to: 


-  Review  program’s  SE  Plan  (SEP)  and  T&E  Master  Plan  (TEMP) 

-  Conduct  program  support  reviews 


•  Portfolio  of  major  acquisition  (ACAT  id  and  iam)  programs, 
supporting  both  Defense  Systems  and  Nil  led  OIPTs: 


Business  Systems 
Communication  Systems 
C2ISR  Systems 
Fixed  Wing  Aircraft 
Unmanned  Systems 


-  Rotary  Wing  Aircraft 

-  Land  Systems 

-  Ships 

-  Munitions 

-  Missiles 


Systems  Engineering  and  T&E  Support  to  Over 
150  Major  Programs  in  Ten  Domain  Areas 
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Driving  Technical  Excellence  into 

Programs 
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SEP  Evaluation  Areas 


•  Structured  approach  with  multiple  perspectives 
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Systems  Engineering  Plan  Activity 

(since  November  2004) 


Programs  submitting  SEPs:  47 

Number  of  SEPs  reviewed:  79 
u  Approved:  12 
u  Pending  final  approval:  1 
u  Pending  draft  review:  10 

Reviews  planned  for  rest  of  FY06:  >100 
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Emerging  SEP  Issues  -  Trends** 

(not  systemic  across  all  programs) 


**BASED  ON  ANALYSIS  OF  47  PROGRAMS 


Trend  Analysis  -  5x5  SEP  Focus  Areas 
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PSR  Evaluation  Areas 


•  Repeatable,  tailorable,  exportable 
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Program  Support  Review  Activity 

\  (since  March  2004) 


PSRs/NARs  completed:  33 
AOTRs  completed:  7 
Nunn-McCurdy  Certification:  3 
Participation  on  Service-led  IRT’s:  4 
Technical  Reviews:  3 
Reviews  planned  for  rest  of  FY06 
«  PSRs/NARs:  12+ 

«  AOTRs:  2 
u  Nunn-McCurdy:  2 
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Programs  by  Domain  Area 
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Representative  Systemic  Issues  (1  of  4) 


1.0  Mission  Capabilities/Requirements 


-  Reliability  requirements  lack  mission  context 

-  Lack  of  growth  margins 

-  Upgrade  programs  lack  measurable  baseline  requirements 

-  Systems  of  Systems  not  well  defined;  Stovepiped  ORDs/CDDs 

-  Requirement  creep  leads  to  systems  engineering  churn 

-  Difficulty  in  balancing  requirements  (e.g.,  transportability,  lethality 
and  survivability  requirements) 


2.0  Resources 

-  Small,  overworked  program  offices 

-  Plans  to  evaluate  joint  interoperability  not  well  defined 
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Representative  Systemic  Issues  (2  of  4) 


3.0  Management 


-  Reluctance  to  demonstrate  key  functionality  in  SDD  phase 
»  Integration  of  Mission  Equipment  Packages  onto  platforms 

-  Success  oriented  schedules  trivialize  integration  risks 
»  COTS  poses  integration  and  support  challenges 

-  Concurrent  development  and  testing  schedules 

-  Lack  of  planning  for  follow-on  increments  and  technical  refresh 

-  Avoidance  of  quantifiable  Milestone  exit  criteria 

-  PMs  not  leveraging  lessons  learned  from  other  programs 

-  Lack  of  overall  SoS  integrator  with  authority  and  resources 
»  Poor  funding  commitment  for  SoS  programs 

»  Lack  of  issue  resolution  process  across  program  and  Service  lines 

-  Poor  communication  across  IPTs 

-  Lack  of  measures-driven  approach  to  risk  management 
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Representative  Systemic  Issues  (3  of  4) 


4.0  Technical  Process 


-  Dependence  on  critical  technologies 

»  Late  Technology  Readiness  Assessments  preclude  ITAs 

-  Technology  Development  phase  not  used  properly  to  mitigate  risks 

-  Lack  of  disciplined  SE  processes  and  SE  reviews,  on  all  programs 

»  No  “time”  to  conduct  full  suite  of  SE  technical  reviews 
»  Insufficient  time  between  SE  technical  reviews 

-  Limited  capability  demonstrated  by  MS  C 

-  Systems  Engineering 

»  Lack  of  disciplined  SE  process,  metrics,  missing  technical  reviews, 
technology  risks  not  mitigated 

-  T&E  Planning 

»  Success  oriented  T&E  schedules;  No  time  for  corrective  actions 
»  Lack  of  attention  to  reliability  growth 
»  Poor  plans  to  mature  suitability  during  SDD  phase 
»  Hesitancy  to  establish  exit  criteria  for  test  phases 
»  Plans  to  evaluate  joint  interoperability  not  well  defined 
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Representative  Systemic  Issues  (4  of  4) 


5.0  Technical  Product 


-  Production  Planning 

»  Production  Readiness  Reviews  (PRRs)  not  always  conducted 
•  PRRs  at  key  suppliers  not  always  planned 
»  Lack  of  supplier  management  plans 
»  Movement  to  improving  processes;  eliminating  waste 

-  Software 

»  Software  processes  not  institutionalized 
»  No  plans  to  apply  lessons  learned  into  successive  builds 
»  Systems  and  spiral  software  requirements  undefined 
»  Software  reuse  strategies  are  inconsistent  across  programs 
»  Software  support  plan  missing 
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Long  Term  Mid  Term  Near  Term 


Systemic  Analysis  Vision  and  Expected 

Outcomes 


Vision:  Illuminate  systemic  program  performance  strengths  and  weaknesses  in 
an  informative  and  consistent  manner  in  support  of  stakeholder  decision 
making  and  more  effective  acquisition  policy,  education  and  training. 


Desired  Outcomes: 
r  Improved  state  of  the  practice 

•  Provide  foundational  information  to  support  policy,  education,  training  and  identification  of  best 
J  practices 

•  Inform  target  audiences  of  issues  and  their  root  causes,  risks  and  recommended  solutions 
based  on  lessons  learned 

^  •  Improve  PSR  process  (e.g.  methodology,  training  for  teams,  templates  etc.) 

r  Informed  decisions 

J  •  Data  to  inform  leadership  decisions,  and  support  leadership  questions 

•  Ability  to  correlate  program  symptoms  (seen  in  OIPTs,  DAES,  etc)  to  systemic  indicators 

^  Parametric  Modeling 

•  Trend  data,  analyzed  over  time 

<  -  Track  individual  cost,  schedule,  performance  over  time 

-  Track  improved  performance  vs.  corrective  actions  made;  successful  practices  incorporated 
Identify  relationships  across  multiple  data  sources  (SEP/TEMP,  DAES,  NARs,  etc.)  and  CAIG 
cost  model 
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Systemic  Analysis  Model 


Steps  1A,  2-4  Underway 


Version 


Providing  Value  Added  Oversight 


•  Tactical,  Program  and  Portfolio  Management 


PEOs  &  PMs... 


•  PSR 

•  AOTR 

•  SEP 

•  TEMP 

•  DAES 


Improved  Program 
Execution  thru... 
Program  Unique 
Recommendations 


Acquisition  Leadership 


Improved  Acquisition  Decision 
Making  thru... 

•  Greater  Program  Transparency 

•  Acquisition  Insight 


Strategic  Management 


DoD  Acquisition  Community 

Improved  Acquisition 

Support  to  Warfighter 

•  Systemic  Issues  &  Risks 

•  Systemic  Strengths  &  Indicators 

Improved  Acquisition 

Decision  Making 

_ 

Recommendations 

y~ 

•  Policy/Guidance  •  Best  Practices 

•  Education  &  Training  •  Other  Processes  (JCIDS,  etc) 

•  Oversight  (DABS/ITAB) 

•  Execution  (staffing) 
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Questions/Discussion 


Contact  Information: 

Dave  Castellano 

OUSD(AT&L)  Defense  Systems 

Deputy  Director,  Assessments  and  Support 

David.Castellano@osd.mil 
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